The present disclosure relates to stream processing, and more specifically, to methods and systems of stream processing including runtime adaptations.
Stream processing applications perform a sequence of transformations on data streams and are composed as data flow graphs, where each vertex of the graph is an operator instance and each edge is a stream connection. In general, stream processing applications execute data transformations upon the arrival of a stream of data items, referred to as a tuples, and sends the newly computed data item to an output stream. In order to achieve high-performance and scalability, the stream processing application can be executed in a distributed fashion over a set of hosts.
Typically, streaming applications are deployed by submitting the composed data flow graph to a target stream processing infrastructure, which continuously runs the application until it is explicitly cancelled. In general, multiple applications can be submitted to the infrastructure at different times. These applications can connect to each other at runtime to form time-evolving solutions.
Streaming applications often need to be adapted to changes in runtime conditions. For instance, when the application is overloaded due to a transient high input data rate, it may need to temporarily apply load shedding policies to maintain a guaranteed quality of service, or response time. However, stream processing languages that are used for developing stream processing applications do not provide constructs for runtime adaptation. This is because these languages are generally declarative. As a result, developers focus on expressing data processing logic, but not adapting to changes in runtime conditions.